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ENHANCED SWITCHOVER FOR MPLS FAST REROUTE 



5 BACKGROUND OF THE INVENTION 

The present invention relates to data networking and more particularly to systems 
and methods for rerouting around failed links and/or nodes. 

The Internet and IP networks in general have become enablers for a broad range 
of business, government, and personal activities. More and more, the Internet is being 
10 relied upon as a general information appliance, business communication tool, 

entertainment source, and as a substitute for traditional telephone networks and broadcast 
media. As the Internet expands its role, users become more and more dependent on 
uninterrupted access. 

To assure rapid recovery in the event of failure of a network link or node, so- 
1 5 called "Fast Reroute" techniques have been developed. In a network employing Fast 
Reroute, traffic flowing through a failed link or node is rerouted through one or more 
preconfigured backup tunnels. The preconfigured backup tunnels facilitate a key goal of 
Fast Reroute techniques, the redirection of interrupted traffic within tens of milliseconds. 
This minimizes impact on the user experience. 

20 The Fast Reroute techniques have been developed in the context of MPLS 

(Multiprotocol Label Switching) where traffic flows through label switched paths (LSPs). 
When an element such as a link or node fails, all of the LSPs using that failed element are 
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redirected through preconfigured backup tunnels that route around the impacted segments 
of the LSPs. These backup tunnels are generally used only for a very short time since 
simultaneously with the rerouting through the backup tunnels, the head ends of all the 
5 affected primary LSPs are notified of the failure. This causes the head ends to reroute the 
primary LSPs around the failures so that the backup tunnels are no longer needed. 

A key aspect of meeting the reliability and quality of service objectives of Fast 
Reroute is the switchover time from the failed segments of the protected primary LSPs to 
their preconfigured backup tunnels. A certain amount of time transpires while the failure 

10 is detected and the actual switchover occurs. In a modern high-speed packet forwarding 
device, the disposition of a received packet is determined by the contents of a compact 
forwarding table stored in high speed memory. The forwarding table's entries include 
so-called "adjacency" information that is used to encapsulate the packet for its transit to 
the next-hop node in addition to one or more labels that are imposed on the packet to 

15 implement the MPLS scheme. Each forwarding equivalence class forwarding over a 
primary LSP will have an entry in such a table. For optimal performance, only a single 
look-up is required to access the contents of this table and this look-up is based on the 
forwarding equivalence class (incoming label or longest match destination address 
prefix). When an LSP's traffic is to be rerouted into a backup tunnel, the switchover is 

20 accomplished by rewriting all of the appropriate entries. 

A problem arises, however, in that a single failed link or node may carry a large 
number of protected LSPs and therefore a large number of entries corresponding to the 
forwarding equivalence classes forwarding over those protected LSPs. A switchover may 
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therefore require rewriting a large number of entries. If, for example, one thousand 
entries must be rewritten in response to the failure of a single link and each entry rewrite 
requires 150 microseconds, the switchover process will consume 150 milliseconds during 
5 which service will be interrupted. This level of delay is unacceptable. 

What is needed are readily scaleable systems and methods for accelerating the 
switchover to preconfigured backup tunnels upon failure of a protected link or node, 
while at the same time not slowing down the normal forwarding process. 
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Summary Of The invention 

Embodiments of the present invention provide a Fast Reroute implementation 
where switchover time to backup tunnels upon failure of a protected network element is 
5 independent of a number of entries corresponding to forwarding equivalence classes 
forwarding over LSPs using that element. During normal operation of a packet 
forwarding device, adjacency information for a received packet is retrieved from a 
forwarding table based on a look-up of the packet's forwarding equivalence class. This 
adjacency information is used to encapsulate and forward the packet to a next-hop node 
10 on the LSP. Upon failure of a link or node, the appropriate entries in this forwarding 
table are rewritten to implement the switchover to preconfigured backup tunnels. 

According the present invention, however, the switchover is effective even before 
the rewrite process has completed. Upon detection of the failure, forwarding processing 
shifts to a fix-up mode. During the fix-up mode, the look-up to the previously mentioned 

15 forwarding table is followed by a look-up to a backup tunnel adjacency table based on a 
pointer retrieved from the forwarding table. The backup tunnel adjacency table includes 
adjacency information to send the packet through a pre-configured backup tunnel. 
Operations relating to use of this backup tunnel adjacency table increase forwarding time 
somewhat but only during the special fix -up mode. Once each entry is rewritten, the fix- 

20 up mode operation can be skipped for that entry. If there are no pending rewrites, fix-up 
mode operation can be skipped entirely. 
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A first aspect of the present invention provides a method for operating a 
forwarding device. The method includes: receiving a packet to be forwarded, identifying 
a first forwarding table entry corresponding to a forwarding equivalence class of the 
5 packet, if a forwarding interface identified by the first forwarding table entry is protected 
by an active backup tunnel and adjacency information for the active backup tunnel has 
not yet been included in the entry, performing a look-up in a second forwarding table to 
retrieve the adjacency information for the active backup tunnel, and forwarding the 
packet in accordance with the adjacency information retrieved from the second 
1 0 forwarding table. 

A second aspect of the present invention provides a method of operating a 
forwarding device. The method includes: providing a forwarding table with entries 
corresponding to primary tunnels. Each of the entries includes: adjacency information 
for packets matching the entry, a backup tunnel table identifier pointing to adjacency 

15 information for a backup tunnel assigned to the entry, and a pointer to a backup tunnel 
active flag for a backup tunnel protecting the corresponding primary tunnel. The method 
further includes: providing a backup tunnel active table holding the backup tunnel active 
flags pointed to by the forwarding table entries, and providing a backup tunnel adjacency 
table with entries indexed by backup tunnel table identifier. Each of the entries includes 

20 adjacency information for packets taking a backup tunnel referenced by the entry's 
backup tunnel identifier. 

A third aspect of the present invention provides a method of operating a 
forwarding device. The method includes: storing adjacency information for a primary 
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tunnel in a first forwarding table, establishing a backup tunnel protecting a segment of the 
primary tunnel, storing adjacency information for the backup tunnel in a second 
forwarding table, when the primary tunnel is operative, forwarding packets assigned to 
5 the primary tunnel based on a single look-up to the first forwarding table, and upon 
activation of the backup tunnel in response to failure of the primary tunnel segment, 
forwarding packets assigned to the primary tunnel via the backup tunnel based on a look- 
up in the first forwarding table followed by a look-up in the second forwarding table. 

Further understanding of the nature and advantages of the inventions herein may 
10 be realized by reference to the remaining portions of the specification and the attached 
drawings. 
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Brief Description Of The drawings 



Fig. 1 depicts a network device according to one embodiment of the present 
invention. 

Fig. 2 depicts a linecard according to one embodiment of the present invention. 

Fig. 3 depicts data structures used in forwarding packets according to one 
embodiment of the present invention. 

Fig. 4 is a flow chart describing steps of reacting to a failure according to one 
embodiment of the present invention. 

Fig. 5 is a flow chart describing steps of forwarding a packet according to one 
embodiment of the present invention. 
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DESCRIPTION OF SPECIFIC EMBODIMENTS 



The present invention will be described with reference to an exemplary label 
switched router (LSR) in a representative network. The LSR may be e.g., an edge router 
5 or interior router of an MPLS network. The present invention, however, can be applied 
to a wide variety of network devices and network configurations. 

At the ingress to a label-switched network, one or more labels are assigned to 
each incoming packet before forwarding the packet to the next-hop node. At each 
intermediate node, a forwarding selection and a new substitute label are determined by 

10 based on the label found in the received packet. At the network egress (or one hop 
prior), a forwarding decision is made based on the incoming label but no label is 
included when the packet is sent on to the next hop. Label switched paths (LSPs) are 
tunnels through the network constructed by the control plane by distributing labels and 
other information used to make forwarding decisions among the network nodes. 

15 Fig. 1 depicts a network device 100 that may be used to implement embodiments 

of the present invention. Network device 100 includes a central processor 102 that 
executes codes stored in a program memory 104. Program memory 104 is one example 
of a computer-readable medium. Program memory 104 can be, e.g., a volatile memory. 
Another form of computer-readable medium storing the same codes would be some type 

20 of non-volatile storage such as floppy disks, CD-ROMS, DVD-ROMS, hard disks, flash 
memory, etc. A carrier wave that carries the code across the network is another example 
of a computer-readable medium. A packet memory 106 is also provided for intermediate 
storage of packets. Network device 100 interfaces with physical media via a plurality of 
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linecards 108. Although only two linecards 108 are shown it is understood that there may 
be numerous linecards within network device 100. It will also be appreciated that each 
linecard 108 may include numerous separate physical interfaces. For example, a 
5 particular linecard may include a large number of electrical and/optical interfaces. Such 
interfaces include, but are not limited to Ethernet interfaces, Gigibit Ethernet interfaces, 
DS-1, DS-3, Frame relay interfaces, ATM interfaces, SONET interfaces, dial-up 
interfaces, DSL interfaces, wireless interfaces, etc. The various components are 
interconnected by a backplane 1 10. 

10 Fig. 2 depicts structure of one of the linecards 108 according to one embodiment 

of the present invention. Representative linecard 108 includes a processor 202 to perform 
operations specified by aspects of the present invention. Like processor 102, processor 
1 10 may execute instructions stored in a program memory 204. Program memory 204 
may represent any of the computer-readable medium types described in reference to 

15 memory 104 and instructions for processor 202 may be stored on any type of volatile or 
fixed computer-readable medium as referred to above. A packet memory 206 provides 
intermediate storage of packets internally within linecard 108. Linecard 108 includes an 
internal interface 208 to backplane 110 of network device 100 and also includes external 
ports 210 to provide connectivity outside network device 100. The precise structure of 

20 external ports 210 will of course depend on the particular type of network, transmission 
medium, etc., to which they are connected. 
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Processor 202 also has access to a fast memory 212. Fast memory 212 may be 
used for high speed look-ups of forwarding information. Fast memory 208 may be, e.g., 
a SRAM, a content-addressable memory, etc., or some combination thereof. 

5 Network device 100 operates in accordance with various protocols including, e.g., 

TCP/IP, MPLS, MPLS Traffic Engineering, MPLS Fast Reroute, etc. The following 
documents (and other documents referenced by these documents) define protocols that 
may be practiced by network device 100. 

E. Rosen, et al., "Multiprotocol Label Switching Architecture," RFC 3031, 
1 0 Internet Engineering Task Force, January 200 1 . 

Braden, et al. "Resource ReSerVation Protocol (RSVP)-Version 1 Functional 
Specification," RFC 2205, Internet Engineering Task Force, September 1997. 

Awduche, et al., "Requirements for Traffic Engineering Over MPLS," RFC 2702, 
Internet Engineering Task Force, September 1999. 

15 Ashwood-Smith, et al., "Generalized MPLS Signaling - RSVP-TE Extensions," 

Internet Draft, Internet Engineering Task Force, October 2001. 

Pan, et al., "Fast Reroute Techniques in RSVP-TE," Internet Draft, Internet 
Engineering Task Force, October 2001. 

The contents of the above documents are herein incorporated by reference in their 
20 entirety for all purposes. 
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In one embodiment, routing functionality is divided among the various linecards 
108 and processor 102. Processor 102 operates as a Routing Processor (RP). Routing 
processor 102 is responsible for control plane functionality. Forwarding of packets is 
5 handled by linecards 108. 

In one implementation, when a packet is received by a particular linecard 108 that 
linecard determines the output linecard and interface and forwards the packet there. Prior 
to transfer of the packet, the receiving linecard encapsulates the packet with any needed 
layer 2 addressing information for forwarding to the next hop node and the label or labels 
10 required by operation of the MPLS and/or MPLS Fast Reroute protocols. The term 

"adjacency information," as used herein refers generally to information such as the layer 
2 source and destination addresses, the linecard and interface to be used for forwarding, 
policy information for a given LSP, etc. "Forwarding information" includes adjacency 
information and the labels to be imposed on the packets traveling via the LSP. 

1 5 Look-up and attachment of layer 2 address information and a label (or labels) to 

the packet may be divided between the ingress linecard and egress linecard in a number 
of ways. However, in the discussion herein, all of the needed information for forwarding 
packets is stored on the ingress card. Layer 2 encapsulation and imposition of any 
needed labels also occur in the ingress card in the discussion herein. The routing 

20 processor (processor 202) sets up label switched paths (LSPs) and backup tunnels by 

appropriately modifying forwarding information on the linecards 108. The primary LSPs 
may support, e.g., Traffic Engineering tunnels, layer 2 or layer 3 virtual private networks, 
pseudowire emulated circuits, virtual private LAN service, etc. 
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If network device 100 is an edge ingress router to the MPLS network, then the 
look-up of forwarding information will be based on the destination IP address prefix of 
the packet since the packets arrive without an MPLS label. For such packets, the 
5 forwarding equivalence class is specified by a variable length address prefix and a mask 
that specifies the length of the prefix within the address. There is a match between the 
destination address and a forwarding equivalence class if the destination address value 
and prefix value are a bit-wise identical for the length of the prefix specified by the mask. 
The forwarding equivalence class of the packet is the one specified by the longest prefix 
10 that matches the packet's destination address. If, on the other hand, network device 100 
is an interior LSR of the MPLS network, the look-up of forwarding information is based 
on a label of the received packet. The label specifies the forwarding equivalence class. 

In the prior art, switchover to a backup tunnel is implemented by rewriting 
appropriate entries in the forwarding tables of the linecards. A single link or node failure, 

15 however, may require numerous such entries to be rewritten. The rewrite process itself 
thus becomes a limiting factor to achieving desired switchover times. According to 
embodiments of the present invention, switchover is accelerated by using a two-stage 
look-up process, but unlike in certain prior art implementations, the two-stage look-up 
process is employed only while the forwarding table is rewritten. In this way, a large 

20 number of LSPs may be accommodated on the same protected link or node without any 
increase of switchover time. Also, unlike prior art schemes that always perform a two- 
stage look-up to accommodate the possibility of active backup tunnels, embodiments of 
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the present invention compromise packet handling performance only during the brief 
intervals required to rewrite forwarding tables. 

Fig. 3 depicts data structures useful in implementing a forwarding scheme 
5 according to embodiments of the present invention. There are three tables. A primary 
forwarding table 302 is the sole source of forwarding information in a normal mode. In a 
special fix-up mode provided by embodiments of the present invention, adjacency 
information may be retrieved instead from a backup tunnel adjacency table 304. Backup 
tunnel adjacency table 304 is indexed by an identifier referred to herein as the backup 

10 tunnel table identifier. The backup tunnel table identifier uniquely identifies a 

combination of backup tunnel and the interface connected to the link or node protected 
by the backup tunnel. Note that the same backup tunnel can protect both a link and a 
node to which the link connects. If this is the case then only one backup tunnel table 
identifier is used because both the link and node are accessible through the same 

15 interface. This identifier scheme recognizes that a particular backup tunnel can be used 
to protect multiple network elements and the forwarding processes described herein 
requires unique identification of both the backup tunnel and protected element to know 
whether the second look-up is in fact necessary for a given LSP. Each entry in table 304 
includes the adjacency information for a particular backup tunnel table identifier. 

20 A backup tunnel active table 306 includes a flag for each backup tunnel table 

identifier. This flag indicates whether the identified backup tunnel is actively protecting 
a link or node accessed via the identified interface. Since backup tunnel active table 306 
requires only a single bit per backup tunnel table identifier, holding the state for 2048 
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backup tunnel table identifiers requires only 64 words of memory. In addition to the 
three depicted tables, there is also a global fix-up flag to indicate whether the linecard is 
currently in the fix-up mode, i.e., whether there are any pending rewrites of the adjacency 
5 information in primary forwarding table 302. 

Table 302 typically includes an entry for each forwarding equivalence class (i.e., 
received label or longest match destination address prefix). Since a single LSP can carry 
traffic of many forwarding equivalence classes, there can be many entries for each LSP. 
Access to table 302 is via a single lookup based on the packet's forwarding equivalence 
10 class. Referring now to the details of a particular entry of primary forwarding table 302, 
several representative fields are shown. A primary tunnel adjacency field 308 includes 
layer 2 source and destination addresses to be used in encapsulating the packet for 
forwarding to the next-hop node. There are three label fields 310, 312, and 314 that hold 
one or more labels to be imposed on packets to be forwarded. 

15 There is a fix-up flag 316 within the third label field 314. The fix-up flag 316 is 

set if forwarding for the entry's LSP has been switched to the associated backup tunnel 
but the information in adjacency field 308 has not yet been rewritten to reflect activation 
of that backup tunnel. The label for the protecting backup tunnel is given in the field 
318. This is the label that will be imposed on packets when they are to travel via the 

20 backup tunnel instead of any label stored in label fields 3 10, 3 12, and 3 14. A backup 

tunnel table identifier field 320 holds a pointer to both backup tunnel active table 306 and 
backup tunnel adjacency table 304. 
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The various data structures of Fig. 3 are preferably stored in fast memory 212 of 
each linecard 108. The adjacency information and labels of primary forwarding table 302 
are determined by the control plane as LSPs are established and modified. The contents 
5 of backup tunnel adjacency table 304 are determined as the control plane preconfigures 
backup tunnels to protect links and/or nodes. The contents of backup tunnel active table 
306 change as backup tunnels are put into operation. 

Fig. 4 is a flowchart describing steps of switching over a primary LSP from a 
failed next segment to a protecting backup tunnel. Linecard 108 reports this failure to 

10 routing processor 202. At step 402, one of linecards 108 detects the failure of a link or 
node to which is it connected. The failure of a particular link or node means that all of 
the primary LSPs that use that link or node will also fail. Routing processor 202 
responds by notifying all of the linecards 108 of the failure. The backup tunnel which 
has been preconfigured to protect the failed LSP segment should now be invoked. At 

15 step 404, each of the linecards 108 sets its global fix-up flag if is not already set to 
indicate that there is now an active rewrite process for the adjacency information in 
primary forwarding table 302. The global fix-up flag may already be set due to other 
Fast Reroute operations. 

Then at step 406, appropriate bits are set in backup tunnel active table 306 to 
20 reflect each combination of active backup tunnel and interface linked to the protected 
element. At this point, the switchover to backup tunnel operation has essentially already 
occurred even though no entries in forwarding table 302 have been rewritten. Execution 
of steps 402, 404, and 406 may require only approximately 5 milliseconds. At step 408, 
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the adjacency information in forwarding table 302 is rewritten with the backup tunnel 
adjacency information. All of the entries corresponding to LSPs impacted by the failure 
will have their entries rewritten. This process occurs in the background in parallel with 
5 normal forwarding operations. As each entry is rewritten, the fix-up flag associated with 
the entry is cleared. This allows fix-up mode operation to be skipped for that entry. 
When all of the affected entries corresponding to a given backup tunnel table identifier 
have been rewritten, the corresponding bit in the backup tunnel active table is cleared. 
Once all of the impacted LSPs have their forwarding table entries rewritten, the global 
10 fix-up flag is cleared at step 410. The flag clearing of step 410 occurs only if there are no 
other pending Fast Reroute rewriting operations pending. 

At the completion of Fast Reroute operation for a given LSP, when the LSP is 
rerouted to avoid the failed element and traffic is taken off the backup tunnel, the 
corresponding entries in primary forwarding table 302 are modified with revised 
15 adjacency information and possibly revised labels to impose. Also, the fix-up flag for the 
entry is initialized as set. This flag is also initialized as set when a new entry in table 302 
is established. 

Fig. 5 is a flow chart describing steps of handling a packet to be forwarded 
according to one embodiment of the present invention. At step 502 a packet to be 
20 forwarded is received. At step 504, the appropriate entry is located in primary 

forwarding table 302 based on the forwarding equivalence class of the packet. If the 
packet has a previously imposed MPLS label or labels, the "topmost" label is used to 
identify the appropriate entry in forwarding table 302. If there is no previously imposed 
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label, the longest match technique is used to find the appropriate entry in table 302 based 
on the packet's destination address. The packet is thus now associated with a particular 
entry in primary forwarding table 302. 

5 A step 506 tests whether the global fix-up flag is currently set indicating that at 

least some entries are currently in fix-up mode with Fast Reroute in effect but the entry 
not yet rewritten. If the global fix-up flag is not set, then processing proceeds to step 508 
where the adjacency information and labels found in the selected entry of primary 
forwarding table 302 are used to encapsulate and forward the packet. In the flow chart of 
10 Fig. 5, step 508 may be reached by a number of paths. Reaching step 508 indicates that 
the relevant LSP is in normal operation or that a backup tunnel is being used to bypass a 
segment of the LSP but the entry adjacency information has already been rewritten to 
reflect use of the backup tunnel. 

If step 506 finds that the global fix-up flag is set, then at step 510 the fix-up flag 
1 5 located in the primary forwarding table entry is tested. It may be that there are pending 
rewrites for other entries but this entry has already been rewritten and thus there is no 
need to refer to backup tunnel adjacency table 304 or backup tunnel active table 306. If 
the fix-up flag is clear, indicating that this entry has already been rewritten (or that the 
pending rewrite operations do not even relate to this LSP) then normal forwarding occurs 
20 at step 508. 

If the fix-up flag for the entry is set, then processing proceeds to a step 512. At 
step 512, the backup tunnel table identifier for the selected entry is retrieved and used as 
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a pointer to a particular bit in backup tunnel active table 306. If this bit is set then there is 
an active backup tunnel protecting the entry's LSP. If the bit is clear, Fast Reroute is not 
active for the LSP and normal forwarding occurs at step 508. If the bit is set, then at step 
5 514, the backup tunnel table identifier of the entry is used as a pointer to backup tunnel 
adjacency table 304. The adjacency information is then retrieved at step 516 and used to 
encapsulate the packet for forwarding along with the backup tunnel label and other labels 
found in the primary forwarding table entry. 

It will be appreciated that fix-up mode involves extra look-ups to backup tunnel 
10 active table 306 and backup tunnel adjacency table 304 and therefore a slowing in packet 
forwarding operations. However, this extra look-up occurs only during fix-up mode and 
only for entries that have not yet been rewritten. Rewrite operations will typically be 
completed in hundreds of milliseconds so the overall performance impact is minimal. 
Also, those of skill in the art will appreciate that routers typically operate well below their 
1 5 maximum packet handling capacity further reducing the impact of fix-up mode. The 
selective use of fix-up mode thus achieves a performance improvement in switchover 
times that is preserved even as the number of entries using protected LSPs using grows. 

It is understood that the examples and embodiments that are described herein are 
for illustrative purposes only and that various modifications and changes in light thereof 
20 will be suggested to persons skilled in the art and are to be included within the spirit and 
purview of this application and scope of the appended claims and their full scope of 
equivalents. 
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